File System Issues Lab

The labs in this module must be run on server1.example.com.

Goals

Recover an ext file system using a backup superblock
Find files across the system that have been modified recently
Recover a corrupted RPM database
Work with immutable files

1. File System Issues - Lab Setup

Discover the device name of the extra disk created in server1.example.com.

# fdisk -l |grep "Disk \/"|grep -v mapper
Disk /dev/vda: 8589 MB, 8589934592 bytes, 16777216 sectors
Disk /dev/vdb: 4294 MB, 4294967296 bytes, 8388608 sectors

/dev/vda is the OS disk!

On /dev/vdb, create a new LV.

# parted -s /dev/vdb mklabel msdos
# parted -s /dev/vdb mkpart primary 0% 100%
# pvcreate /dev/vdb1
# vgcreate extra /dev/vdb1
# lvcreate -n disk extra -l 100%FREE

Create an ext4 file system on /dev/extra/disk and mount it in /mnt/temp.

# mkfs.ext4 -b 2048 /dev/extra/disk
# mkdir /mnt/temp
# mount /dev/extra/disk /mnt/temp/

Copy the contents of /etc from the system to /mnt/temp, then unmount the file system.
```
# rsync -aPv /etc/ /mnt/temp
# umount /mnt/temp
```

Simulate corruption of the EXT superblock.

# dd if=/dev/urandom of=/dev/extra/disk bs=2K count=260

Verify that the superblock is corrupt (you should see an error with this command).
```
# dumpe2fs /dev/extra/disk
```
Use a mkfs in non write mode (-n) with the same original block size (-b) to get a list of backup superblocks.
```
# mkfs.ext4 -n -j -b 2048 /dev/extra/disk
```
Use e2fsck to repair the file system using one of the superblocks found in the previous step.
```
# e2fsck -b 16384 -y /dev/extra/disk
```
Mount the repaired file system and see if the files copied to it still exist and are not corrupted, some files may end up in lost+found:
```
# mount /dev/extra/disk /mnt/temp/
# ls /mnt/temp/
# md5sum /etc/passwd /mnt/temp/passwd
# ls /mnt/temp/lost+found
```
- If files did end up in lost+found, they will most likely lose their filenames and be renamed to an inode number.
- You can use the file command against the lost+found files to determine what kind of file they are.
  # file /mnt/temp/lost+found/*
- You can also look at the contents of the file if it is a text file to determine what it is or use md5sum to compare it with similar files on a working system.

2. File Auditing

You suspect a system has been hacked. Determine which files have been active (accessed, created, or modified) in the last 15 minutes.
```
# find /etc -cmin -15
```
- Some of these files are expected to change, but not all. In practice, you would take a close look at the files returned to make sure they have not been compromised.
Find files in standard binary directories (/usr, /usr/bin, /sbin, /usr/sbin, etc.) that may have an invalid user (not in /etc/passwd).
```
# find {,/usr{,/local}}/{,s}bin -nouser
```
- These may be false positives if the files found are owned by LDAP users and LDAP connectivity is down.
Files that are SUID root can be potential security risks. Note that there are many valid SUID root files. Compare output with a healthy system.
```
# find / -perm /4000 -user root -ls
```
A user was detected on the system that should not exist. The UID for the user was 1001. Find all of the files on the entire system that this user owned.
```
# find / -uid 1001
```
Use find in conjunction with RPM to see if there are any files that exist that are not part of any installed packages.
```
# find {,/usr{,/local}}/{,s}bin | xargs rpm -q --whatprovides | grep "not owned"
```

3. Package Auditing

To see which packages are currently installed using yum:
```
# yum list installed
```
To see a chronological listing of yum package installations, updates, and removals see /var/log/yum.log.
```
# less /var/log/yum.log
```
Alternatively, rpm also tracks package installations in its database.
```
# rpm -qa --last
```

4. Corrupt `rpm` Database

Before removing the rpm database lock files, use the lsof command to make sure no running processes are using them.
```
# lsof | grep /var/lib/rpm
```
Remove the stale rpm database lock files.
```
# rm -rf /var/lib/rpm/__db*
```

Back up the flat rpm metadata.

# tar -cjvf /root/rpmdb-$(date +%Y%m%d-%H%M).tar.bz2 /var/lib/rpm

Verify package metadata.

# cd /var/lib/rpm
# /usr/lib/rpm/rpmdb_verify Packages

If corrupt, (assume it is for the lab), make a dump of what is retrievable.

# mv Packages Packages.bad
# /usr/lib/rpm/rpmdb_dump Packages.bad|/usr/lib/rpm/rpmdb_load Packages
# /usr/lib/rpm/rpmdb_verify Packages

Rebuild the rpm DB using the new dump.
```
# rpm -v --rebuliddb
```

5. Immutable Attribute

Set the immutable bit on a file.

# touch /tmp/testfile
# lsattr /tmp/testfile
-------------e- /tmp/testfile
# chattr +i /tmp/testfile
# lsattr /tmp/testfile
----i--------e- /tmp/testfile

Find files with the immutable attribute set.

# lsattr -R / 2> /dev/null | grep '^[^/]...i'

Un-set the immutable bit on a file.
```
# chattr -i /tmp/testfile
```